System and method for data quality monitors

ABSTRACT

Systems and methods are presented for data quality monitoring. Data quality monitors may be created and configured to identify objects with specified data quality issues and/or property values. Objects identified by a data quality monitor can be presented to users for confirmation and resolution. Properties used by the data quality monitor to match objects may also be displayed to users.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

This application is a continuation of Ser. No. 14/094,418, filed Dec. 2,2013, which is a continuation of U.S. patent application Ser. No.13/935,861 filed Jul. 5, 2013, which issued as U.S. Pat. No. 8,601,326.Each of these applications are hereby incorporated by reference hereinin their entireties.

TECHNICAL FIELD

The present disclosure relates to systems and techniques for dataquality monitoring. More specifically, identifying and resolving dataquality issues and presenting data with potential quality issues foruser confirmation and resolution.

BACKGROUND

Traditional automated data integration may result in various dataquality issues. Automatic scripts to search for and resolve such dataquality issues may create further data quality problems.

SUMMARY

There is a need to identify and monitor data quality problemsautomatically and intelligently. In accordance with one aspect, acomputer system is configured to include one or more computer processorsand a tangible storage device storing one or more module configured forexecution by the one or more computer processors in order to: receive,by the computer system, selection of an object type to monitor forpossible data quality problems; receive, by the computer system,selection of one or more data quality criterion; determine potentialdata quality problems by scanning data associated with a plurality ofobjects of the selected object type in order to determine if theselected one or more data quality criterion are matched by respectiveobjects; generate a user interface indicating the potential data qualityproblems of respective objects match the selected one or more dataquality criterion, and including one or more properties of the matchingobjects; receive from a user of the computing environment an indicationof how to resolve the potential data quality problems; and implement theindicated resolution.

In some embodiments, the data quality monitor type is one of: possibleduplicate objects, missing properties, multi-valued properties, unparsedproperties, disallowed enumerations, numeric range violations, and daterange violations. In some embodiments, the data quality monitor type ispossible duplicate objects, and said determining objects havingpotential data quality problems further comprises identifying respectiveobjects each having common property values for a predetermined qualityof the selected one or more properties. In some embodiments, theindication of how to resolve the potential data quality problemcomprises an indication of two or more objects having potential dataquality problems that are duplicates; and said implementing theindicated resolution comprises combining the two or more indicatedobjects into a single object. In some embodiments, the one or moreproperties of the objects having potential data quality problemsindicates a quantity of the identified respective objects sharing theselected one or more properties. In some embodiments, the data qualitymonitor type is missing properties; and said determining objects havingpotential data quality problems further comprises identifying respectiveobjects that are missing a predetermined of the selected one or moreproperties.

In some embodiments, the data quality monitor type is multi-valuedproperties; and said determining objects having potential data qualityproblems further comprises identifying respective objects that havemultiple values for ones of the selected one or more properties. In someembodiments, the indication of how to resolve the potential data qualityproblem comprises an indication of one of the multiple values that iscorrect; and said implementing the indicated resolution furthercomprises removing all of the multiple values except for the indicatedone or multiple values that is correct. In some embodiments, the dataquality monitor type is unparsed properties; and said determiningobjects having potential data quality problems further comprisesidentifying respective objects that have property values for one or moreof the selected one or more properties including unparsed data.

In some embodiments, the data quality monitor type is disallowedenumerations; and said determining objects having potential data qualityproblems further comprises identifying respective objects that haveproperty values for one or more of the selected one or more propertiesincluding disallowed enumerations. In some embodiments, the data qualitymonitor type is numeric range violation; and said determining objectshaving potential data quality problems further comprises identifyingrespective objects that have property values that violate an allowednumeric range for the respective selected property. In some embodiments,the data quality monitor type is date range violation; and saiddetermining objects having potential data quality problems furthercomprises identifying respective objects that have property values thatviolate an allowed date range for the respective selected property. Insome embodiments, the computer system further comprises receivingselection of one or more object types that the selected object type isrequired to be associated with in order to be included in the scannedplurality of objects. In some embodiments, the computer system furthercomprises receiving one or more Boolean operators for the selected oneor more properties and/or an indication of how many of the selected oneor more properties are required in order to identify the respectiveobjects as objects having potential data quality problems.

In accordance with another aspect, a computer implemented methodcomprises under control of a computing system having one or morephysical processors and configured to process large amounts of data,receiving, by the computing system, selection of an object type tomonitor for possible data quality problems; receiving, by the computingsystem, selection of a data quality monitor type; receiving, by thecomputing system, selection of one or more properties of objects of theselected object type; determining objects having potential data qualityproblems by scanning data associated with a plurality of objects of theselected object type in order to locate respective objects matching theselected one or more properties; generating a user interface indicatingthe objects having potential data quality problems, and including one ormore properties of the objects having potential data quality problems;receiving, by the computing system, an indication of how to resolve thepotential data quality problems; and implementing the indicatedresolution. In some embodiment, the data quality monitor type is one of:possible duplicate objects, missing properties, multi-valued properties,unparsed properties, disallowed enumerations, and numeric rangeviolation, and date range violation. In some embodiment, the computerimplemented method further comprises receiving one or more Booleanoperators for the selected one or more properties and/or an indicationof how many of the selected one or more properties are required in orderto identify the respective objects as objects having potential dataquality problems.

In accordance with another aspect, a non-transitory computer-readablestorage medium storing computer-executable instructions configured todirect a computing system to: receive selection of an object type tomonitor for possible data quality problems; receive selection of one ormore data quality criterion; determine potential data quality problemsby scanning data associated with a plurality of objects of the selectedobject type in order to determine if the selected one or more dataquality criterion are matched by respective objects; generate a userinterface indicating the potential data quality problems of respectiveobjects; match the selected one or more data quality criterion, andincluding one or more properties of the matching objects; receive from auser of the computing environment an indication of how to resolve thepotential data quality problems; and implement the indicated resolution.In some embodiments, the the data quality monitor type is one of:possible duplicate objects, missing properties, multi-valued properties,unparsed properties, disallowed enumerations, and numeric rangeviolation, and date range violation. In some embodiments, thenon-transitory computer-readable storage medium further comprisesreceiving selection of one or more object types that the selected objecttype is required to be associated with in order to be included in thescanned plurality of objects.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates one embodiment of a database system using anontology.

FIG. 2 illustrates one embodiment of a process of generating personobjects with properties from multiple data sources.

FIG. 3 illustrates one embodiment of a system for creating data in adata store using a dynamic ontology.

FIG. 4A illustrates a sample user interface using relationshipsdescribed in a data store using a dynamic ontology.

FIG. 4B illustrates a user interface that is configured to receiveuser-selected criteria for a data quality monitor, according to oneembodiment.

FIG. 5A illustrates an example interface that is configured to enableusers to establish criteria for a duplicate objects data qualitymonitor.

FIG. 5B illustrates potentially duplicate objects identified by a dataquality monitor system.

FIG. 5C illustrates an example interface that is configured to enableusers to review additional data related to identified potentiallyduplicate objects and determine whether the objects are duplicate andshould be resolved.

FIG. 5D illustrates an example user interface wherein potentiallyduplicate objects are resolved and results of the resolution aredisplayed.

FIG. 6 is a flowchart depicting an illustrative operation ofidentifying, confirming, and removing or resolving duplicate objects.

FIG. 7 is a flowchart depicting an illustrative process of creating andrunning data quality monitors based on received data quality criteria.

FIG. 8A illustrates an example user interface that allows a user toselect criteria for a data quality monitor.

FIG. 8B illustrates another example user interface wherein a user hasconfigured a potentially duplicate person monitor to search on bothperson objects and case objects.

FIG. 8C illustrates an example user interface wherein a group ofpossibly duplicate person objects are displayed and properties of theseperson objects are shown.

FIG. 9 illustrates an example user interface wherein a user has selecteda missing properties monitor and is provided with example objectproperties to select for monitoring.

FIG. 10 illustrates an example user interface that displays multiplecase objects that were identified in an executed missing propertiesmonitor.

FIG. 11 illustrates a computer system with which certain methods andmodules discussed herein may be implemented.

DETAILED DESCRIPTION OF CERTAIN EMBODIMENTS

Definitions

In order to facilitate an understanding of the systems and methodsdiscussed herein, a number of terms are defined below. The terms definedbelow, as well as other terms used herein, should be construed toinclude the provided definitions, the ordinary and customary meaning ofthe terms, and/or any other implied meaning for the respective terms.Thus, the definitions below do not limit the meaning of these terms, butonly provide exemplary definitions.

Ontology: Stored information that provides a data model for storage ofdata in one or more databases. For example, the stored data may comprisedefinitions for object types and property types for data in a database,and how objects and properties may be related.

Database: A broad term for any data structure for storing and/ororganizing data, including, but not limited to, relational databases(Oracle database, mySQL database, etc.), spreadsheets, XML files, andtext file, among others.

Data Object or Object: A data container for information representingspecific things in the world that have a number of definable properties.For example, a data object can represent an entity such as a person, aplace, an organization, a market instrument, or other noun. A dataobject can represent an event that happens at a point in time or for aduration. A data object can represent a document or other unstructureddata source such as an e-mail message, a news report, or a written paperor article. Each data object may be associated with a unique identifierthat uniquely identifies the data object. The object's attributes (e.g.metadata about the object) may be represented in one or more properties.

Object Type: Type of a data object (e.g., person, event, or document).Object types may be defined by an ontology and may be modified orupdated to include additional object types. An object definition (e.g.,in an ontology) may include how the object is related to other objects,such as being a sub-object type of another object type (e.g. an agentmay be a sub-object type of a person object type), and the propertiesthe object type may have.

Properties: Attributes of a data object that represent individual dataitems. At a minimum, each property of a data object has a property typeand a value or values.

Property Type: The type of data a property is, such as a string, aninteger, or a double. Property types may include complex property types,such as a series data values associated with timed ticks (e.g. a timeseries), etc.

Property Value: The value associated with a property, which is of thetype indicated in the property type associated with the property. Aproperty may have multiple values.

Link: A connection between two data objects, based on, for example, arelationship, an event, and/or matching properties. Links may bedirectional, such as one representing a payment from person A to B, orbidirectional.

Link Set: Set of multiple links that are shared between two or more dataobjects.

Data Quality Monitors: rules including one or more criterion configuredto identify potential data quality problems in data objects or betweenmultiple data objects. Actions to confirm possibly data quality problemsand/or correct such data quality problems may be taken by a user or byautomatic processes.

Object Centric Data Model

To provide a framework for the following discussion of specific systemsand methods described herein, an example database system 210 using anontology 205 will now be described. This description is provided for thepurpose of providing an example and is not intended to limit thetechniques to the example data model, the example database system, orthe example database system's use of an ontology to representinformation.

In one embodiment, a body of data is conceptually structured accordingto an object-centric data model represented by ontology 205. Theconceptual data model is independent of any particular database used fordurably storing one or more database(s) 209 based on the ontology 205.For example, each object of the conceptual data model may correspond toone or more rows in a relational database or an entry in LightweightDirectory Access Protocol (LDAP) database, or any combination of one ormore databases.

FIG. 1 illustrates an object-centric conceptual data model according toan embodiment. An ontology 205, as noted above, may include storedinformation providing a data model for storage of data in the database209. The ontology 205 may be defined by one or more object types, whichmay each be associated with one or more property types. At the highestlevel of abstraction, data object 201 is a container for informationrepresenting things in the world. For example, data object 201 canrepresent an entity such as a person, a place, an organization, a marketinstrument, or other noun. Data object 201 can represent an event thathappens at a point in time or for a duration. Data object 201 canrepresent a document or other unstructured data source such as an e-mailmessage, a news report, or a written paper or article. Each data object201 is associated with a unique identifier that uniquely identifies thedata object within the database system.

Different types of data objects may have different property types. Forexample, a “person” data object might have an “Eye Color” property typeand an “Event” data object might have a “Date” property type. Eachproperty 203 as represented by data in the database system 210 may havea property type defined by the ontology 205 used by the database 209.

Objects may be instantiated in the database 209 in accordance with thecorresponding object definition for the particular object in theontology 205. For example, a specific monetary payment (e.g., an objectof type “event”) of US$30.00 (e.g., a property of type “currency”)taking place on Mar. 27, 2009 (e.g., a property of type “date”) may bestored in the database 209 as an event object with associated currencyand date properties as defined within the ontology 205.

The data objects defined in the ontology 205 may support propertymultiplicity. In particular, a data object 201 may be allowed to havemore than one property 203 of the same property type. For example, a“person” data object might have multiple “Address” properties ormultiple “Name” properties.

Each link 202 represents a connection between two data objects 201. Inone embodiment, the connection is either through a relationship, anevent, or through matching properties. A relationship connection may beasymmetrical or symmetrical. For example, “person” data object A may beconnected to “person” data object B by a “Child Of” relationship (where“person” data object B has an asymmetric “Parent Of” relationship to“person” data object A), a “Kin Of” symmetric relationship to “person”data object C, and an asymmetric “Member Of” relationship to“Organization” data object X. The type of relationship between two dataobjects may vary depending on the types of the data objects. Forexample, “person” data object A may have an “Appears In” relationshipwith “Document” data object Y or have a “Participate In” relationshipwith “Event” data object E. As an example of an event connection, two“person” data objects may be connected by an “Airline Flight” dataobject representing a particular airline flight if they traveledtogether on that flight, or by a “Meeting” data object representing aparticular meeting if they both attended that meeting. In oneembodiment, when two data objects are connected by an event, they arealso connected by relationships, in which each data object has aspecific relationship to the event, such as, for example, an “AppearsIn” relationship.

As an example of a matching properties connection, two “person” dataobjects representing a brother and a sister, may both have an “Address”property that indicates where they live. If the brother and the sisterlive in the same home, then their “Address” properties likely containsimilar, if not identical property values. In one embodiment, a linkbetween two data objects may be established based on similar or matchingproperties (e.g., property types and/or property values) of the dataobjects. These are just some examples of the types of connections thatmay be represented by a link and other types of connections may berepresented; embodiments are not limited to any particular types ofconnections between data objects. For example, a document might containreferences to two different objects. For example, a document may containa reference to a payment (one object), and a person (a second object). Alink between these two objects may represent a connection between thesetwo entities through their co-occurrence within the same document.

Each data object 201 can have multiple links with another data object201 to form a link set 204. For example, two “person” data objectsrepresenting a husband and a wife could be linked through a “Spouse Of”relationship, a matching “Address” property, and one or more matching“Event” properties (e.g., a wedding). Each link 202 as represented bydata in a database may have a link type defined by the database ontologyused by the database.

FIG. 2 illustrates one embodiment of a process of generating personobjects with properties from multiple data sources. Problems related todata quality may arise in certain of the person data objects as a resultof the automatic integration of data from multiple sources.Additionally, errors may be introduced by data entry mistakes, such astypographical errors, and/or in other manners. In the example of FIG. 2,multiple data sources provide data to the data federation system 225including utility bills 215, phone records 217, tickets 217, driver'sregistration 221, and other personal data 223, etc. Depending on theembodiment, the data sources that provide these various types of datamay be different entities or a single entity.

Each data record (e.g., data provided about a person from one of thedata sources) may include information about one or more persons, andproperties of person objects from the various data sources may vary. Forexample, the utility bills data 215 may include properties such as name,address, account number, amount due, past utility usage information,past payment history, and/or minimum amount due, while the phone recordsdata 217 may include information such as name, address, phone number(s)and/or account number(s) (which may or may not be the same as phonenumber), amount due, usage information, past payment history, minimumamount due, phone calls placed, message sent, recipient's information(phone number), information about each phone call placed and/or eachmessage sent, for example. The tickets data 219 may include informationsuch as name of a driver, address, phone number, driver license number,vehicle registration number, event involved, ticket type, and/or date ofthe ticket. The driver's registration data 221 may include informationsuch as name, date of birth, height, weight, eye color, hair color, datelicense issued, and/or expiration date.

In some embodiments, through data federation system 225, data fromvarious sources may be automatically federated to create multiple personobjects 227. For example, the utility bills data 215 of a person mayshow that the person lives at 123 Oak Street, and his phone number is123-456-7890. The same person's phone records data 217 may show that helives at 789 Willow Road and his phone number is 098-765-4321. Theperson's tickets data 219 may show that this person lives at 123 OakStreet, his gender is “female,” he was born on 01/13, and his phonenumber is 098-765-4321. Finally, the person's driver's registration 221may show that he lives at 789 Willow Road, he was born on Jan. 13, 1967,his gender is “male,” and his phone number is 123-456-7890.

In this example, the data federation system 225 may update thecorresponding person object (or create the person object if one doesn'talready exist) using some automated data federation rules and processes.In some embodiments, the end result of the data federation process maybe something that simply takes all the source information and compilesit together in the person object. For example, if the data federationprocess integrates the data from various sources directly without anychanges, the person may be indicated as having two genders: male andfemale. The person may also be shown as having two birth dates: “01/13”and “Jan. 13, 1967.” Furthermore, the first birth date of “01/13” may beinterpreted as January 2013 instead of January 13^(th) of a year.

Accordingly, due to data quality issues, the federated data through thedata federation system 225 may involve all kinds of discrepancies,missing values, and other types of potential problems. These problemsmay sometimes be difficult to recognize such that the problems may becorrected.

FIG. 3 is a block diagram illustrating exemplary components and datathat may be used in identifying and storing data according to anontology. In this example, the ontology may be configured, and data inthe data model populated, by a system of parsers and ontologyconfiguration tools. In the embodiment of FIG. 3, input data 300 isprovided to parser 302. The input data may comprise data from one ormore sources, such as data sources that provide the various data itemsdiscussed with reference to FIG. 2. For example, an institution may haveone or more databases with information on credit card transactions,rental cars, and people. The databases may contain a variety of relatedinformation and attributes about each type of data, such as a “date” fora credit card transaction, an address for a person, and a date for whena rental car is rented. The parser 302 is able to read a variety ofsource input data types and determine which type of data it is reading.

In accordance with the discussion above, the example ontology 205comprises stored information providing the data model of data stored indatabase 209, and the ontology is defined by one or more object types310, one or more property types 316, and one or more link types 330.Based on information determined by the parser 302 or other mapping ofsource input information to object type, one or more data objects 201may be instantiated in the database 209 based on respective determinedobject types 310, and each of the objects 201 has one or more properties203 that are instantiated based on property types 316. Two data objects201 may be connected by one or more links 202 that may be instantiatedbased on link types 330. The property types 316 each may comprise one ormore base types 320, such as a string, number, etc. In some embodiments,property types include only a base type of string, number, date,enumeration, or composite. Depending on the embodiment, property typesmay or may not have a hierarchical structure like objects do. Forinstance, a missing child object type can be a sub-type of the personobject type, but a home property type cannot be a sub-type of a locationproperty type.

In some embodiments, composite property types contain multiplecomponents, but may still be considered a single property. For instance,an Address property may be a composite property, formed from multiplecomponents, such as Address 1 (string), Address 2 (string), City(string), State (string), and ZIP (could be number or a string).

In some embodiments, the ontology rules may also restrict whichproperties are allowed on which objects, as well as what linkrelationships are allowed between objects.

In an embodiment, a user of the system uses an object type editor 324 tocreate and/or modify the object types 310 and define attributes of theobject types. In an embodiment, a user of the system uses a propertytype editor 326 to create and/or modify the property types 316 anddefine attributes of the property types. In an embodiment, a user of thesystem uses link type editor 328 to create the link types 330.Alternatively, other programs, processes, or programmatic controls maybe used to create link types and property types and define attributes,and using editors is not required.

In an embodiment, creating a property type 316 using the property typeeditor 326 involves defining at least one parser definition using aparser editor 322. In some embodiments, a parser definition comprisesmetadata that informs parser 302 how to parse input data 300 tonormalize the value assigned to the property type 316 that is associatedwith the parser definition. For example, the ontology parser maydetermine how to convert both inputs “Jan. 1, 2002” and “Jan. 1, 2002”into a normalized value “2002-01-01 00:00.”

Depending on the embodiment, a composite property parser may identifyindividual components. For example, a composite parser for may turn anaddress “111 Main St., New City, ST, 12345” into normalized values suchas: Address 1: 111 Main St; City: New City; State: ST; Zip: 12345. Insome embodiments, if parsing rules fail, the original value may be keptin an unparsed format. This may be later identified as a potential dataquality issue by a data quality monitor. In some embodiments, codeparsers may choose which property type to use. In some embodiments, thecode may map a data column from the input to a property type in theontology and then parse the input.

In an embodiment, each parser definition may comprise a regularexpression parser 304A or a code module parser 304B. In otherembodiments, other kinds of parser definitions may be provided usingscripts or other programmatic elements. Once defined, both a regularexpression parser 304A and a code module parser 304B can provide inputto parser 302 to control parsing of input data 300.

Using the data types defined in the ontology, input data 300 may beparsed by the parser 302 determine which object type 310 should receivedata from a record created from the input data, and which property types316 should be assigned to data from individual field values in the inputdata. Based on the object-property mapping 301, the parser 302 selectsone of the parser definitions that is associated with a property type inthe input data. The parser parses an input data field using the selectedparser definition, resulting in creating new or modified data 303. Thenew or modified data 303 is added to the database 209 according toontology 205 by storing values of the new or modified data in a propertyof the specified property type. As a result, input data 300 havingvarying format or syntax can be created in database 209. The ontology205 may be modified at any time using object type editor 324, propertytype editor 326, and link type editor 328, or under program controlwithout human use of an editor. Parser editor 322 enables creatingmultiple parser definitions that can successfully parse input data 300having varying format or syntax and determine which property typesshould be used to transform input data 300 into new or modified inputdata 303.

The properties, objects, and the links (e.g. relationships) between theobjects can be visualized using a graphical user interface (GUI). Forexample, FIG. 4A displays a user interface showing a graphrepresentation 403 of relationships (including relationships or links404, 405, 406, 407, 408) between the data objects (including dataobjects 421, 422, 423, 424, 425, 426) that are represented as nodes inthe example of FIG. 4. In this embodiment, data objects 421-426 areperson objects, such as the person objects 227 discussed with referenceto FIG. 2. In this example, the person nodes (associated with persondata objects) may have relationships to other person nodes, for example,through payment objects. For example, relationship 404 is based on apayment associated with the individuals indicated in person data objects421 and 423. The link 404 represents these shared payments (for example,the individual associated with data object 411 may have paid theindividual associated with data object 413 on three occasions). Theserelationships may be stored as links, or in some embodiments, asproperties, where a relationship may be detected between the properties,or as event objects. In some cases, as stated above, the links may bedirectional. For example, a payment link may have a direction associatedwith the payment, where one person object is a receiver of a payment,and another person object is the payer of payment.

In addition to visually showing relationships between the data objects,the user interface may allow various other manipulations. For example,the objects within database 209 may be searched using a search interface450 (e.g., text string matching of object properties), inspected (e.g.,properties and associated data viewed), filtered (e.g., narrowing theuniverse of objects into sets and subsets by properties orrelationships), and statistically aggregated (e.g., numericallysummarized based on summarization criteria), among other operations andvisualizations.

FIG. 4B illustrates an object filter user interface 460 that isconfigured to receive user-selected criteria for a data quality monitorand display results given the criteria, according to an embodiment ofthe present disclosure.

In the embodiment of FIG. 4B, the object filter initially receives fromthe user a selection of one or more types of objects to be monitored. Inthis example, the selected object type is “Case,” and therefore caseobjects satisfying the specified criteria are displayed. In general, acase is an object type that refers to multiple other objects of possiblymultiple object types. A case object may refer to various types ofinformation, such as a case related to a criminal investigation, a caseof West Nile virus infection, a case involving litigants, or a caserelated to economic analysis. Depending on specific user requirements, acase object may include various properties such as names of peopleinvolved, dates of events, address, phone number, outcome, etc., and acase object may be associated with multiple other object types, such asone or more person objects, event objects, etc. In some otherembodiments, other object types may be selected, such as person, event,vehicle, etc.

In this example, criterion (e.g., required object types or values ofproperties) to be included in the data quality monitor can be added byclicking the add criterion button 461. For example, the monitor mayinitially include no criterion (or possibly default criterion in oneembodiment), and the user adds a first criterion 469 by clicking on theadd criterion button 461, and subsequently adds the criterion 471 and473 by clicking on the add criterion button 461 additional times.

The user interface of FIG. 4B allows the user to select Booleanoperators to further define requirements of the monitor. For example,the user has indicated in interface 463 that the object type is requiredto be a “case” and has indicated in interface 465 that any one of thelisted criterion 469, 471, 473 is sufficient for a match. In oneembodiment, these interfaces 463, 467 also allow the user to indicate aminimum number of the criterion that must be met, rather than theindicated “any” (in interface 467) or “all” (in interface 463). In otherembodiments, relationships between the criteria may be provided in othermanners.

As shown in FIG. 4B, criterion may reference any available properties ofthe selected object type. For example, the criteria of “category =8D” isselected in criterion 469 and “category =8F” is selected in criterion471, while criterion 473 refers to a particular date. A user may defineany additional or different criterion based on any other properties ofthe objects to be monitored, such as location =Boston, Mass., airline=Southwest, date range=Feb. 14, 2009 to Mar. 9, 2011, etc.

In some embodiments, a data quality monitor may be configured torestrict selected objects to objects connected to one or more specificobject or objects connected to one or more type of objects. For example,if “person” is an object type that is already selected, a user may addan icon representing an object type (such as “case” 475) next to the“person” object type 479, and add a link between the two object types.With the added link between the case object 475 and the person object479, the monitor is configured to select only case objects that are alsoconnected to a person object. This feature may be useful to furtherlimit the scope of searched objects. For example, if a user is onlyinterested in case objects connected to a known criminal A, in additionto satisfying the criteria on the right side of the panel, the selectedcase objects are also related to the criminal A.

Data Quality Monitors

FIG. 5A illustrates an example user interface 510 that is configured toenable users to establish criteria for a data quality monitor.Initially, the user may select an existing monitor (such as a monitorthat was previously generated by the user and/or another user) using thedrop-down menu 512. Alternatively, the user can create a new monitor byselecting create new monitor button 514. If the user creates a newmonitor, the example user interface 510 allows the user to provide acustom name, such as “Duplicate persons” by inputting the name in a textbox. In addition, in some embodiments, the user interface 510 may allowthe user to publish a custom monitor by clicking the publish monitorbutton 520, which makes the monitor available to other users (e.g., viathe select monitor drop-down menu 512) such as to perform the same dataquality monitoring on other datasets.

In the embodiment of FIG. 5A, the user can select one or more particularobject types to monitor. In this example, person objects 522 relating tocase objects 524 are configured to be monitored by this data qualitymonitor. A user may further edit the objects to monitor by clicking the“edit” button 521, which opens a user interface such as the interfaceshown in FIG. 4B. In some embodiments, this may mean that person objectsbeing monitored should be related to one or more case object, and personobjects not related to any case objects should not be monitored. In someembodiments, only person objects 522 may be configured to be monitoredwithout the additional requirement that the person objects 522 should berelated to one or more case objects 524.

In this example, the user has selected a duplicate objects data qualitymonitor using the monitor type drop-down menu 526. Data quality monitortypes may be directed to other data quality issues, such as such asmissing values, unparsed properties, numeric range violations,properties with values that do not match allowed enumerations, and othertypes of potential data quality issues, which may also be available inthe menu 526. The user interface 510 further provides a list ofavailable property types associated with the chosen object type (in thisexample, person object type) for which the monitor will search.

Furthermore, the data quality monitor may also be configured to allow auser to input a number of minimum matches among the provided propertycriteria that result in a matching person object (or other object typein other implementations). For example, if a user has specified 4properties and a minimum match of 2 (as shown in the example of FIG.5A), matching at least two of the specified properties between twoperson objects would be deemed sufficient for identifying those twoobjects as potentially duplicate objects. In the example of FIG. 5A, thefollowing properties are selected to be matched on: name, date of birth,Social Security Number, and license number.

In some embodiments, the properties to match on in order to determinewhether objects are duplicate may be added or deleted. A user may click“Add property” button 538 to add additional properties. A user may alsoclick on the remove property icon 540 next to a property to remove thatproperty. Property settings may also be saved with the monitor.

In the embodiment of FIG. 5A, results of the data quality monitor 510may be obtained by clicking on the “Run Monitor Now” button 530.Depending on the embodiment and the specific dataset used, duplicateobjects may or may not be found according to the established dataquality monitor criteria. In this example, 3 groups of person objectsare found to be possible duplicates according to the specified criteria.Depending on the embodiment, results of the data quality monitor may beviewed in a graph setting by selecting 534, or in a browser setting byselecting 536.

FIG. 5B illustrates potentially duplicate objects identified by the dataquality monitor defined in user interface 510 of FIG. 5A. In thisexample, two person objects, Bad Person 1 540 and Bad Person 2 542 areidentified by the “Duplicate persons” data quality monitor as possiblythe same as each other. In some embodiments where there are many objectsidentified, the objects may be displayed in a graph view or in a browserview with fewer objects shown at first. A user will be given the choiceto drill-down, drill-up, and/or otherwise explore the identifiedobjects.

FIG. 5C illustrates an example interface 544 that is configured toenable users indicate whether the objects are duplicate and should beresolved, possibly after reviewing additional data related to identifiedpotentially duplicate objects (see FIG. 8C, for example). In someembodiments, a different interface may be presented for resolving groupsof potentially duplicate objects. In the example interface 544, a usermay be presented with a question such as “Are you sure you want toresolve these 2 objects?” In some other embodiments, a similar ordifferent question may be asked to communicate to the user that a choicemay be made to resolve the potential data quality issue. In thisexample, a user may choose the default title “Bad Person 2.” If thischoice is made, the title of the new merged objects is Bad Person 2. Inthis embodiment, the data on both Bad Person 1 and Bad Person 2 aremerged to form the new merged object titled Bad Person 2.

In some embodiments, when a potential data quality issue is resolved,the merged object is maintained in the data store and the individualobjects used to create it no longer show up separately in searches. Insome embodiments, the user that initiates object resolution may alsoindicate that the resolution should be effective only in data accessedby the particular user and/or a particular group of user. Thus, theresolution (and/or other changes to data objects) may be selectivelypushed out to other users.

FIG. 5D illustrates an example user interface 546 wherein potentiallyduplicate objects are resolved and results of the resolution aredisplayed. In this example, Bad Person 2 is found to be a potentiallyduplicate of some other objects. Information regarding properties of BadPerson 2, which now include properties of Bad Person 1 that was resolvedinto Bad Person 2, may be displayed. In some embodiments, onlyinformation that is actually used to match duplicate objects or identifyother potentially data quality issues is displayed. In some otherembodiments, all or relevant properties of a selected object may bedisplayed to a user. In this example, Bad Person 2 is show to have thename of “Bad Person 1” and a Social Security Number of 123-45-6789.

FIG. 6 is a flowchart depicting an illustrative operation ofidentifying, confirming, and removing or resolving duplicate objectsaccording to one embodiment. The method of FIG. 6 may be performed by acomputing system used by an analyst (or other user), one or more serversystems that store the object ontology (e.g., database system 210 ofFIG. 1), and/or any other suitable computing system.

The method 600 begins at block 605, where a data quality monitorautomatically determines possible duplicate objects based on a set ofcriteria (e.g., those set in user interface 510 of FIG. 5A). In someembodiments, the set of criteria may be created by a user. In some otherembodiments, the set of criteria may have been already created before(e.g., by the same user, another user, or as a system default) andstored in the data quality monitor. The set of criteria may includeproperty criteria, such as the properties to match and the minimumnumber of properties that needs to be matched. While a possibleduplicates data quality monitor is discussed with reference to FIG. 6,other types of monitors may be executed in a similar manner. Asdiscussed above, the other types of data quality monitors may includemissing values, unparsed properties, numeric range violations,properties with values that do not match allowed enumerations, and othertypes of potential data quality issues.

The method 600 proceeds to block 610, where the data quality monitorcreates associations between possible duplicate objects. For example, iftwo person objects are deemed to be possible duplicates of each otherbased on identical Social Security Number (SSN) and identical date ofbirth (DOB) (where SSN and DOB are selected properties to be matched onand the monitor requires 2 or less matching properties), a “possibleduplicate objects” association/link between these two person objects maybe created. In some situations, the two person objects may have someother properties that are not identical, such as different eye colors.

In another example, if four Ticket objects have the same date ofissuance, name, incident type, driver's license number, and vehiclelicense numbers, the data quality monitor may deem them to bepotentially duplicates (based on the established monitor criteria) andcreate associations/links between the four Ticket objects, identifyingthem as potential duplicates.

At block 615, the data quality monitor provides an indication ofpossible duplicate objects to a user. In some embodiments, theindication that some objects may potentially be duplicates of each othermay be presented in a graph view, wherein the one or more identifiedobjects are displayed and connections between or among the identifiedobjects indicate their associations. A user may use the informationprovided in such a graph view and/or one or more related histogram views(see, e.g., FIG. 8C), for example, to determine whether the objects areduplicates or not. In some other embodiments, the connections between oramong the identified objects may be shown in an explorer view, wherein auser may click on an icon that represents an object to find out moredetails about why certain objects are connected to each other and whatmakes the objects possible duplicates of one another. The createdassociation may indicate the identical matched properties, or possiblyonly the properties that are deemed crucial.

At decision block 720, the data quality monitor receives input from theuser indicating whether or not a set of potential duplicates really areduplicates. For example, the user may review additional properties ofidentified possible duplicate objects in order to make thedetermination. The user may provide the input in various manners, suchas the user dragging and dropping duplicate objects onto one another,responding to a query from the system requesting confirmation ofduplication in illustrated objects, and/or by any other manner. In someembodiments, the data quality monitor may be configured to automaticallymake a choice by default in case a user is not available or decides notto perform the confirmation step. In such instances, the data qualitymonitor may automatically make a determination and resolve and/or removeany duplication.

If the user finds that the identified objects are duplicates, theprocess 600 proceeds to block 625, where the duplication issue isresolved. As discussed above, in one embodiment duplicate objects may beresolved by combining the objects into a single object, havingproperties from each of the two or more determined duplicate objects. Inother embodiments, the duplicate objects may remain separate objects,but associations between or among the identified objects may be updatedto indicate the user's confirmation of the duplication. For example, theassociation among the identified objects may be updated to “confirmedduplicates” from “potential duplicates.”

If, however, the user decides that the identified objects may not beduplicates at decision block 620, the process 600 proceeds to block 630,where the indication of possible duplication may be removed by thesystem. In some embodiments, association between or among the identifiedobjects may also be removed or updated to indicate the use'sconfirmation of no duplication. For example, the association among theidentified objects may be updated to “no duplication-confirmed” from“potential duplicates.”

FIG. 7 is a flowchart depicting an illustrative process of creating andrunning data quality monitors based on received data quality criteria,according to one embodiment. The method of FIG. 7 may be performed by acomputing system used by an analyst (or other user), one or more serversystems that store the object ontology (e.g., database system 210 ofFIG. 1), and/or any other suitable computing system.

The process 700 begins at block 705, where the data quality monitorreceives selection of object types. In some embodiments, availableobject types may be presented directly to a user, such as in a drop-downmenu. In some other embodiments, the user can choose more than oneavailable object types. For example, as previously discussed, a personobject type may be selected.

The process 700 proceeds to block 710, where the data quality monitorreceives a selection of required properties of objects to be included inan analysis set. For example, if the selected object type in block 705is a person object, the selection of required properties for the personobjects to be included in the analysis set may include any property ofperson objects, such as the following, without limitation: name,address, phone number, date of birth, Social Security Number, driver'slicense, country of citizenship, immigration status, occupation, etc. Insome cases, the selection of required properties of objects may includea smaller number of properties that are the most crucial to a user. Forexample, if a user wants to run a data quality monitor on case objectsrelated to stolen vehicles, the properties associated with the caseobjects may include Vehicle Identification Number (VIN), date on whichthe vehicle was stolen, location where the vehicle was last seen,vehicle model, vehicle make, and vehicle color, etc. In some othercases, the selection of required properties may include a large numberof properties, depending on the user's requirement. In addition, theobject type selected for monitoring may be required to be associatedwith one or more other object types, such as the required link betweenperson object types and case object types in the example of FIG. 5A.

At block 715, the data quality monitor may receive a selection of dataquality criteria to apply to objects in the analysis set. Depending onthe embodiment, available data quality criteria may first be determinedbased on a selected data quality monitor type, such as (1) duplicateobjects; (2) multi-valued properties; (3) missing properties; (4)unparsed properties; (5) properties with values that do not matchallowed enumerations; (6) numeric range violation, etc. Each dataquality monitor type may include certain criteria for identifyingobjects with potential data quality problems. Additionally, the user cancustomize the data quality criteria that are included in the monitor.Discussed below are examples of certain data quality monitors that maybe selected by a user.

A duplicate objects monitor may identify potentially duplicate objectsthat have one or more identical (or substantially similar) propertyvalues such that they may be considered duplicates. However, objects maynot always be considered potentially duplicate just because they sharesome of the same property values. For example, several person objectsmay share the same mailing address, phone number, and vehicleinformation. In some situations, having such identical property valuesdoes not make the person objects identical because these people may livein the same household. In some other cases, sharing important propertyvalues may result in the system finding that some objects arepotentially duplicates. For example, if several person objects share thesame Social Security Number, phone number, date of birth, then theymight be considered duplicates. In some embodiments, these personobjects may still be considered not potentially duplicates due to otherproperty values that are available or fraud concerns. Thus, depending onthe particular data objects and corresponding properties that are beingmonitored, the user may have different criteria for a duplicate objectsmonitor.

A multi-valued properties monitor may identify objects with propertieswith more than one value. For example, the President of a foreigncountry should be only one person. If the value of the Presidentproperty of that country object has more than one person, this couldindicate a potential data quality problem. In another example, a personobject has two Social Security Numbers. Normally, a person should onlybe associated with only one Social Security Number. However, there couldbe a fraud investigation, and this particular person object has beenassociated with several Social Security Numbers that he has fraudulentlyused. Thus, the person object may be flagged as a potential data qualityproblem in a multi-valued properties monitor.

A missing properties monitor may identify objects with property valuesmissing and/or missing property types. For example, in some embodiments,a blood donor database may include many person objects, each having aproperty called “blood type.” Because it may be important to determinethe blood donors' blood types, a person object having no blood typeproperty value or a person object having no blood type property at allmay potentially be problematic.

An unparsed properties monitor may identify objects associated withunprocessed and/or incomplete information in an origin database. Forexample, a field agent in a law enforcement agency may have input thefollowing information regarding a suspect (e.g., a person data object):“loc: CA.” Afterwards, if no more information is provided by the fieldagent, it may not be very clear whether this information means“location=California,” “location=Canada” or something else. In anotherexample, a suspect may have provided a date of birth. However, thesuspect's date of birth may use a different calendar year system, suchas the lunar-year calendar system or the Tibetan calendar system.Therefore, if the suspect's lunar birth date is Jan. 20, 1986, thesuspect's real birth date may be a different date. Property values assuch may also result in data quality issues.

In another example, a date may be provided in a format such asDay/Month/Year (e.g., 13/7/92). A system that mistook the “Day” for“Month” may fail to parse the date of 13/7/92 because there is no month13. The unparsed properties monitor may catch dates that fail to parseunder the Month/Day/Year convention and alert a user to correct the dateparsing rule so that a date provided under the Day/Month/Year formatwill be parsed correctly.

Depending on the embodiment, an enumerated property may be a propertywith a restricted or limited set of allowed values. Properties withvalues that do not match allowed enumerations may result from wrongvalues that do not correspond to the property types. For example, aperson object may have an “age” property with a value of “female” and a“gender” property with a value of “55 years old.” In another example,“fmale” may be provided instead of “female” for gender, which may resultin data parsing failures. In some situations, a user may mistakenlyunderstand the acceptable values of a property. For example, a user maybelieve that the location property may have two values such as “CONUS”(meaning Continental U.S.) and “OCONUS (outside Continental U.S.) whenin fact the location property has values such as “bar,” “park,” or“home.” The data quality monitor may detect such discrepancies and alerta use to correct the problems accordingly. In some other situations, theproblem could be less obvious. For example, a case object may have a“stolen vehicle” property with a value of “Escape at Lincoln St.” Thefield agent who originally entered the information may have meant thatthe vehicle was stolen while it was on Lincoln St., and the stolenvehicle is a Ford Escape. However, since the information has not beenparsed properly, it may present an unparsed properties issue. Dependingon embodiments and user requirements, such discrepancies in propertyvalues may be detected by a data quality monitor using parametersspecified by the user and/or parameters set up by the system.

A numeric range violation monitor may identify objects with propertyvalues outside of an expected range for the particular property. Forexample, a person object may have an “age” property with a value of“−10” or “450.” Both are clearly wrong. More subtle mistakes can also bedetected. For example, if the system has knowledge that a certainstreet's street numbers are 0 to 100, then the system may be able tofind an erroneous street address value with a street number of 150. Insome other embodiments, a data range violation monitor may also beprovided. For example, if a provided date is invalid or is out ofcontext (e.g., a credit card transaction date in the 14^(th) Century),the data quality monitor may catch such problems and alert the user.

Once a data quality monitor type is selected and criteria of the monitorare established, the process 700 proceeds to block 720, where the dataquality monitor is executed. For example, the data quality monitor mayscan the analysis set for objects matching the specified data qualitycriteria and/or a set of system default criteria.

At block 725, results of the execution of data quality monitor areprovided. In some embodiments, such results are provided in a graphicaluser interface, listing all the objects and/or their specific valuesthat may have violated the defined data quality criteria. In some otherembodiments, such results may be provided in a graphical user interfacethat connects the objects in order to allow the users to see how suchobjects are connected. For example, as previously discussed, twopotentially duplicate objects may be connected by an association named“possibly same as,” showing that the two objects are deemed to bepossible duplicates. Depending on the embodiment, the graphical userinterface may also include property values showing why the identifiedobjects are included. In some other embodiments, the results may also bestored in a file such as a comma-delimited file, an Excel® file, an XMLfile, an HTML file, etc., with information such as name of the objectsand/or property values. In some embodiments, the results may beavailable for users to download.

At block 730, a user may be given the choice of saving the data qualitymonitor for later use. If the user chooses to do so, in someembodiments, the results of several rounds of data quality monitoringmay also be compared to see whether the same data quality issues tend topersist, and whether some data quality problems have already beenresolved. In some other embodiments, a user may not only save dataquality monitors, but also publish them so that other users of thesystem may also run similar or the same searches to detect data qualityissues. Depending on the embodiment, other users may use a publisheddata quality monitor and customize the monitor for their own specificrequirements.

FIG. 8A illustrates an example user interface 840 that allows a user tocreate, load, and/or update criteria for a data quality monitoraccording to one embodiment. The example user interface 840 includeschoices of creating a new monitor 853 or selecting an existing dataquality monitor 850. In some embodiments, if a user chooses to create anew data quality monitor 853, the user interface 840 may presentchoices/questions to the user so that the user may specify criteria tobe included in the new data quality monitor. If a user chooses to selectan existing data quality monitor 850, criteria of the selected dataquality monitor are loaded into the system for execution. The exampleuser interface 840 allows the user to further customize an existing dataquality monitor and save the customized data quality monitor as a newmonitor. In some other embodiments, a data quality monitor may beconfigured such that it can only be customized or edited by certainusers.

The user interface 840 includes options that allow a user to configurethe objects to monitor 855. In this example, the objects selected to bemonitored are case objects. Other types of objects can also bemonitored. Moreover, in some embodiments, a user may edit objects sothat more than one type of objects may be monitored. In someembodiments, even if monitoring is only performed on one type of object,data quality monitoring criteria may require the monitored object to beassociated with one or more other types of objects. For example, a caseobject may be required to be associated with at one or more eventobjects and/or person objects that are of interest to a user.

The example user interface 840 further includes a monitor type selectioncontrol 860 that allows selection of a data quality monitor type. Aspreviously discussed, available data quality monitor types may include:(1) potentially duplicate objects; (2) multi-valued properties; (3)missing properties; (4) unparsed properties; (5) properties with valuesthat do not match allowed enumerations; (6) numeric range violation,and/or any other monitor. In the example of FIG. 8A, potentiallyduplicate objects 860 is selected. Once a data quality monitor type isselected, specific properties of the selected objects to be monitoredmay be provided by the user. Customization of properties is discussedfurther with reference to the example of FIG. 8B.

In some embodiments, data quality monitor settings may also beconfigured to discover orphaned objects, such as objects with no links.Data quality monitor settings may also be limited to run the searchcriteria specified, e.g., in 855, exclusively without checking foranything else.

FIG. 8B illustrates another example user interface 1100 wherein a userhas configured a potentially duplicate person monitor to identify personobjects (linked to case objects) that are potentially duplicates. Thus,as displayed in the area 1125, person objects linked to case objects areconfigured to be monitored. The required association of person objectsto case objects may be a user's desire to monitor only person objectsthat have some connection to a criminal investigation (e.g., “case”).

In the drop-down menu 1130 in this example, the selected data qualitycriterion is potentially duplicate objects (potentially duplicateperson) and the minimum number of properties satisfying the propertyrequirements is selected as two at box 1135.

In this example, the user has selected properties to match on as: name1140, date of birth 1145, Social Security Number 1150, and licensenumber 1155. Therefore, according to the settings in user interface1100, the objects found to be potentially duplicates should have atleast two identical property values, which can be any two propertiesfrom the four selected properties. Thus, for this particular monitor,two objects sharing the same name and Social Security Number would beconsidered possible duplicates and may be presented to the user forfurther confirmation and/or resolution.

As discussed previously, objects matching the criteria specified in adata quality monitor may be displayed in the user interface 1100. Inthis example, a total of three groups of objects are found, as indicatedin 1160. The eleven identified person objects are all related to a caseobject(s) per the link requirement indicated in area 1125. In thisexample, the eleven located person objects are displayed in interface1165, along with connections between the identified person objects.

FIG. 8C illustrates an example user interface 870 wherein a group ofpossibly duplicate person objects are displayed and properties of theseperson objects are shown to indicate properties of the objects, whichmay be useful to the user in determining whether the objects really areduplicates. In this example, a group of five person objects (e.g., ofthe three groups of objects located in the example of FIG. 8B) aredisplayed and an entity properties histogram 871 is illustrated to showvarious properties of the identified objects. As shown at 874, the fiveperson objects have the name “James Bond” or names very similar to“James Bond,” such as “James B.” Furthermore, some of the identifiedperson objects share other properties. In this example, all five personobjects have the same social security number, as shown in 878. Four ofthe five objects have the same date of birth (at 872) and three of theperson objects share the same last name “Bond” (at 876). Finally, allfive identified person objects in this example have at least threeproperties (as required by a user by specifying minimum matches=3) thatmatch another person object. Given the information provided about theidentified properties, the user may be able to confirm whether any orall of the identified objects should be considered duplicates or not. Ifa user confirms that some of the identified objects are in factduplicates, he or she may take further actions such as resolving theduplication and/or updating the associations among person objects.

FIG. 9 illustrates an example user interface 900 wherein a user hasselected a missing properties monitor and is provided with objectproperties to select for monitoring. In this example, the objects beingmonitored are case objects. As previously discussed, missing propertiesmay include data quality issues resulting from absent property valuesassociated with one or more properties and/or missing property typesaltogether. In this example, the user interface 900 includes a list ofproperties associated with the case object so that a user may choosewhich properties and/or property values to monitor.

In this example, the list of properties for user selection in thedrop-down menu 910 includes: (1) location name; (2) location type; (3)address; (4) address city; (5) address state; (6) crime category. Inthis example, the user has chosen to monitor “location name” in order toidentify case objects missing either the property “location name” orobjects with no value associated with the “location name” property. Theuser interface 900 also allows the user to choose to monitor more caseproperties for missing properties, such as by selecting the addcriterion button 845. In some embodiments, other properties associatedwith a case object may be presented for user selection. In some otherembodiments, instead of presenting the properties that a user may chooseto monitor in a drop-down menu, other types of user interface elementsmay be used.

FIG. 10 illustrates an example user interface 1000 that displaysmultiple case objects that were identified in an executed missingproperties monitor. In this example, multiple properties associated withan object have been selected for inclusion in the missing propertiesmonitor. In particular, in addition to a “location name” property in910, “subject victim status” is selected in 1005 and Agent Name isselected in 1007.

Depending on the embodiment, the user interface 1000 may also includeone or more Boolean logic operator between or among the propertiesselected by a user. For example, a user may be able to specify that shewants to monitor case objects missing both the “location name” property(or property value) and the “subject victim status” property (orproperty value). In another example, a user may specify that she onlywants to monitor case objects missing either the “location name”property (or property value) or the “subject victim status” property (orproperty value). The user interface 1000 allows the users to specify aminimum number of matches in box 1135 or some other user interfaceelements. In this example, a minimum of two matches is required and,thus, objects located by this monitor match at least two of the threespecified properties.

In a similar manner as discussed above with reference to FIGS. 8-9, forexample, data quality monitors of other types, such as unparsedproperties, may present a list of available properties for users tochoose from in customizing the monitors. In an unparsed propertiesexample, one or more Boolean logic operators may be provided to specifywhether all the properties selected by the user are required to haveunparsed properties or only one or a subset of the properties arerequired to have unparsed properties.

The user interface 1000 may also include options for users to runmonitor now 1010 or create nightly monitor 1015. If a user chooses tocreate nightly monitors, the user interface 1000 may in some embodimentsask further questions such as the specific time that the configured dataquality monitor should be executed and how the user wish to review thedata quality monitor results. If a user chooses to run the configureddata quality monitor now, the user interface 1000 may send commands toexecute the data quality monitor. After execution, the user interface1000 may display a quantity of objects matching the data quality monitorand/or display the matched objects.

In the example of FIG. 10, 1257 case objects were found as shown at1020. The user interface may present a choice to analyze the matchedobjects using an Object Explorer by choosing 1025. In some embodiments,the user interface 1000 may also include choices for users to view thematched objects in a graph by choosing 1030 or in a browser by choosing1035.

Depending on the embodiment, the user interface 1000 may also displaythe objects matching the configured data quality settings in theexecuted data quality monitor. In this example, the user interfaceincludes an area 865 displaying case objects matching the data qualitysettings. The first six case objects 1040 are displayed as case 1through 6. A different default number of objects may be displayed by theuser interface 1000 as needed or according to a different user setting.

Implementation Mechanisms

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, server computer systems, portable computersystems, handheld devices, networking devices or any other device orcombination of devices that incorporate hard-wired and/or program logicto implement the techniques.

Computing device(s) are generally controlled and coordinated byoperating system software, such as iOS, Android, Chrome OS, Windows XP,Windows Vista, Windows 7, Windows 8, Windows Server, Windows CE, Unix,Linux, SunOS, Solaris, iOS, Blackberry OS, VxWorks, or other compatibleoperating systems. In other embodiments, the computing device may becontrolled by a proprietary operating system. Conventional operatingsystems control and schedule computer processes for execution, performmemory management, provide file system, networking, I/O services, andprovide a user interface functionality, such as a graphical userinterface (“GUI”), among other things.

For example, FIG. 11 is a block diagram that illustrates a computersystem 800 upon which an embodiment may be implemented. Computer system800 includes a bus 802 or other communication mechanism forcommunicating information, and a hardware processor, or multipleprocessors, 804 coupled with bus 802 for processing information.Hardware processor(s) 804 may be, for example, one or more generalpurpose microprocessors.

Computer system 800 also includes a main memory 806, such as a randomaccess memory (RAM), cache and/or other dynamic storage devices, coupledto bus 802 for storing information and instructions to be executed byprocessor 804. Main memory 806 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 804. Such instructions, whenstored in storage media accessible to processor 804, render computersystem 800 into a special-purpose machine that is customized to performthe operations specified in the instructions.

Computer system 800 further includes a read only memory (ROM) 808 orother static storage device coupled to bus 802 for storing staticinformation and instructions for processor 804. A storage device 810,such as a magnetic disk, optical disk, or USB thumb drive (Flash drive),etc., is provided and coupled to bus 802 for storing information andinstructions.

Computer system 800 may be coupled via bus 802 to a display 812, such asa cathode ray tube (CRT) or LCD display (or touch screen), fordisplaying information to a computer user. An input device 814,including alphanumeric and other keys, is coupled to bus 802 forcommunicating information and command selections to processor 804.Another type of user input device is cursor control 816, such as amouse, a trackball, or cursor direction keys for communicating directioninformation and command selections to processor 804 and for controllingcursor movement on display 812. This input device typically has twodegrees of freedom in two axes, a first axis (e.g., x) and a second axis(e.g., y), that allows the device to specify positions in a plane. Insome embodiments, the same direction information and command selectionsas cursor control may be implemented via receiving touches on a touchscreen without a cursor.

Computing system 800 may include a user interface module to implement aGUI that may be stored in a mass storage device as executable softwarecodes that are executed by the computing device(s). This and othermodules may include, by way of example, components, such as softwarecomponents, object-oriented software components, class components andtask components, processes, functions, attributes, procedures,subroutines, segments of program code, drivers, firmware, microcode,circuitry, data, databases, data structures, tables, arrays, andvariables.

In general, the word “module,” as used herein, refers to logic embodiedin hardware or firmware, or to a collection of software instructions,possibly having entry and exit points, written in a programminglanguage, such as, for example, Java, Lua, C or C++. A software modulemay be compiled and linked into an executable program, installed in adynamic link library, or may be written in an interpreted programminglanguage such as, for example, BASIC, Perl, or Python. It will beappreciated that software modules may be callable from other modules orfrom themselves, and/or may be invoked in response to detected events orinterrupts. Software modules configured for execution on computingdevices may be provided on a computer readable medium, such as a compactdisc, digital video disc, flash drive, magnetic disc, or any othertangible medium, or as a digital download (and may be originally storedin a compressed or installable format that requires installation,decompression or decryption prior to execution). Such software code maybe stored, partially or fully, on a memory device of the executingcomputing device, for execution by the computing device. Softwareinstructions may be embedded in firmware, such as an EPROM. It will befurther appreciated that hardware modules may be comprised of connectedlogic units, such as gates and flip-flops, and/or may be comprised ofprogrammable units, such as programmable gate arrays or processors. Themodules or computing device functionality described herein arepreferably implemented as software modules, but may be represented inhardware or firmware. Generally, the modules described herein refer tological modules that may be combined with other modules or divided intosub-modules despite their physical organization or storage

Computer system 800 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 800 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 800 in response to processor(s) 804 executing one or moresequences of one or more instructions contained in main memory 806. Suchinstructions may be read into main memory 806 from another storagemedium, such as storage device 810. Execution of the sequences ofinstructions contained in main memory 806 causes processor(s) 804 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “non-transitory media,” and similar terms, as used hereinrefers to any media that store data and/or instructions that cause amachine to operate in a specific fashion. Such non-transitory media maycomprise non-volatile media and/or volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks, such as storage device810. Volatile media includes dynamic memory, such as main memory 806.Common forms of non-transitory media include, for example, a floppydisk, a flexible disk, hard disk, solid state drive, magnetic tape, orany other magnetic data storage medium, a CD-ROM, any other optical datastorage medium, any physical medium with patterns of holes, a RAM, aPROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge, and networked versions of the same.

Non-transitory media is distinct from but may be used in conjunctionwith transmission media. Transmission media participates in transferringinformation between nontransitory media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 802. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 804 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 800 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 802. Bus 802 carries the data tomain memory 806, from which processor 804 retrieves and executes theinstructions. The instructions received by main memory 806 may retrievesand executes the instructions. The instructions received by main memory806 may optionally be stored on storage device 810 either before orafter execution by processor 804.

Computer system 800 also includes a communication interface 818 coupledto bus 802. Communication interface 818 provides a two-way datacommunication coupling to a network link 820 that is connected to alocal network 822. For example, communication interface 818 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 818 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN (or WAN component tocommunicated with a WAN). Wireless links may also be implemented. In anysuch implementation, communication interface 818 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 820 typically provides data communication through one ormore networks to other data devices. For example, network link 820 mayprovide a connection through local network 822 to a host computer 824 orto data equipment operated by an Internet Service Provider (ISP) 826.ISP 826 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 828. Local network 822 and Internet 828 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 820and through communication interface 818, which carry the digital data toand from computer system 800, are example forms of transmission media.

Computer system 800 can send messages and receive data, includingprogram code, through the network(s), network link 820 and communicationinterface 818. In the Internet example, a server 830 might transmit arequested code for an application program through Internet 828, ISP 826,local network 822 and communication interface 818.

The received code may be executed by processor 804 as it is received,and/or stored in storage device 810, or other non-volatile storage forlater execution.

Each of the processes, methods, and algorithms described in thepreceding sections may be embodied in, and fully or partially automatedby, code modules executed by one or more computer systems or computerprocessors comprising computer hardware. The processes and algorithmsmay be implemented partially or wholly in application-specificcircuitry.

The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure. In addition, certain method or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto can be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically disclosed, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe disclosed example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the disclosed example embodiments.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

Any process descriptions, elements, or blocks in the flow diagramsdescribed herein and/or depicted in the attached figures should beunderstood as potentially representing modules, segments, or portions ofcode which include one or more executable instructions for implementingspecific logical functions or steps in the process. Alternateimplementations are included within the scope of the embodimentsdescribed herein in which elements or functions may be deleted, executedout of order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved, as would be understood by those skilled in the art.

It should be emphasized that many variations and modifications may bemade to the above-described embodiments, the elements of which are to beunderstood as being among other acceptable examples. All suchmodifications and variations are intended to be included herein withinthe scope of this disclosure. The foregoing description details certainembodiments of the invention. It will be appreciated, however, that nomatter how detailed the foregoing appears in text, the invention can bepracticed in many ways. As is also stated above, it should be noted thatthe use of particular terminology when describing certain features oraspects of the invention should not be taken to imply that theterminology is being re-defined herein to be restricted to including anyspecific characteristics of the features or aspects of the inventionwith which that terminology is associated. The scope of the inventionshould therefore be construed in accordance with the appended claims andany equivalents thereof.

What is claimed is:
 1. A computer-implemented method, performed by acomputing system having one or more hardware computer processors and oneor more non-transitory computer readable storage device storing softwareinstructions executable by the computing system to perform thecomputerized method comprising: receiving, via a user interface,selection, from a plurality of object types including entity and event,a first object type and a related object type different than the firstobject type; receiving, via the user interface, selection of a criterionselected from a plurality of criterion that are selectable via the userinterface, wherein the selected criterion is configured to identify dataobjects of the first object type associated with a data object of therelated object type and having a potential problem; identifying, basedon evaluation of at least the selected criterion with reference to aplurality of data objects stored in one or more data stores: a firstdata object of the first object type, the first data object having aplurality of properties, wherein the first data object is identifiedbased on satisfaction of the selected criterion by one or moreproperties of the first data object; and a second data object of therelated object type having a link to the first data object; and causingpresentation, in the user interface, of a graphical representation ofthe first and second data objects as visually linked.
 2. Thecomputer-implemented method of claim 1, wherein the potential problemcomprises at least one of: possible duplicate objects, missingproperties, multi-valued properties, unparsed properties, disallowedenumerations, numeric range violations, or date range violations.
 3. Thecomputer-implemented method of claim 1, wherein the criterion comprisesa value for a property.
 4. A non-transitory computer storage mediumstoring computer executable instructions that when executed by acomputer hardware processor perform operations comprising: receiving,via a user interface, selection, from a plurality of object typesincluding entity and event, a first object type and a related objecttype different than the first object type; receiving, via the userinterface, selection of a criterion selected from a plurality ofcriterion that are selectable via the user interface, wherein theselected criterion is configured to identify data objects of the firstobject type associated with a data object of the related object type andhaving a potential problem; identifying, based on evaluation of at leastthe selected criterion with reference to a plurality of data objectsstored in one or more data stores: a first data object of the firstobject type, the first data object having a plurality of properties,wherein the first data object is identified based on satisfaction of theselected criterion by one or more properties of the first data object;and a second data object of the related object type having a link to thefirst data object; and causing presentation, in the user interface, of agraphical representation of the first and second data objects asvisually linked.
 5. The non-transitory computer storage medium of claim4, wherein determining the second data object further comprisesidentifying a first property of the first data object that matches asecond property of the second data object.
 6. The non-transitorycomputer storage medium of claim 5, wherein the potential problemcomprises a possible duplicate object, and the operations furthercomprise: receiving a user selection to resolve the potential problem;generating a merged data object including a property from at least oneof the first or second data objects; removing at least one of the firstor second data objects from a data store; and storing the merged dataobject in the data store.
 7. The non-transitory computer storage mediumof claim 4, wherein the operations further comprise: generating a dataquality monitor including the first object type and the criterion; andstoring the data quality monitor in a data store, wherein the dataquality monitor is executable to identify potential data problems.
 8. Acomputer system comprising: one or more hardware computer processorsprogrammed, via executable code instructions, to: receiving, via a userinterface, selection, from a plurality of object types including entityand event, a first object type and related object type different thanthe first object type; receiving, via the user interface, selection of acriterion selected from a plurality of criterion that are selectable viathe user interface, wherein the selected criterion is configured toidentify data objects of the first object type associated with a dataobject of the related object type and having a potential problem;identifying, based on evaluation of at least the selected criterion withreference to a plurality of data objects stored in one or more datastores; a first data object of the first object type, the first dataobject having a plurality of properties, wherein the first data objectis identified based on satisfaction of the selected criterion by one ormore properties of the first data object; and a second data object ofthe related object type having a link to the first data object; andcausing presentation, in the user interface, of a graphical,representation of the first and second data objects as visually linked.9. The computer system of claim 8, wherein identifying the first dataobject of the first object type with the potential problem furthercomprises matching the criterion with a property of the first dataobject.
 10. The computer system of claim 8, wherein determining thesecond data object further comprises identifying a first property of thefirst data object that matches a second property of the second dataobject.
 11. The computer system of claim 8, wherein the potentialproblem comprises a possible duplicate object, and the one or morehardware computer processors are further programmed to: receive a userselection to resolve the potential problem; and generate a merged dataobject including a property from at least one of the first or seconddata objects.
 12. The computer system of claim 11, wherein the one ormore hardware computer processors are further programmed to: remove atleast one of the first or second data objects from a data store.
 13. Thecomputer system of claim 11, wherein the one or more hardware computerprocessors are further programmed to: store the merged data object in adata store.
 14. The computer system of claim 10, wherein the one or morehardware computer processors are further programmed to: generate a dataquality monitor including the first object type and the criterion; andstore the data quality monitor in a data store, wherein the data qualitymonitor is executable to identify potential problems.